% scribe: Percy Liang
% lastupdate: 24 October 2005
% lecture: 15
% references: Durrett, Section 2.3
% title: Characteristic Functions: Inversion Formula
% keywords: characteristic functions, uniqueness theorem, Parseval's identity, normal distribution, exponential distribution, Cauchy distribution, Fubini's theorem, Dominated Convergence Theorem, weak convergence
% end

\documentclass[12pt, letterpaper]{article}

\include{macros}

\begin{document}

\lecture{15}
{Characteristic Functions: Inversion Formula}
{Percy Liang}{pliang@cs.berkeley.edu}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

These notes were partially adapted from the notes of Bo Li and Ankit Jain
from previous years.  References: \cite{durrett}, section 2.3.

\section{Introduction}
% keywords: characteristic function, uniqueness theorem, Parseval's identity
% end

Recall that the characteristic function of a random variable $X$
with distribution $\P$ is defined as follows:
\begin{equation}
\label{eqn:charfn}
\varphi_X(t) = \E e^{itX} = \int e^{itx} \P(dx), \quad t \in \R.
\end{equation}

Given the distribution of $X$,
we can compute the characteristic function of $X$
using equation~\ref{eqn:charfn}.
But if we know the characteristic function $\varphi_X(t)$,
how do we recover the distribution of $X$?

The Uniqueness Theorem tells us that there is exactly one distribution of $X$
with the given characteristic function $\varphi_X(t)$.
We will derive an inversion formula
that explicitly tells us this distribution in terms of $\varphi_X(t)$.

The main tool we will use to derive the inversion formula is Parseval's identity:
\begin{equation}
\label{eqn:parseval}
\int \varphi_X(y) \Q(dy) = \varphi_Y(x) d\P(dx)
\end{equation}

\begin{proof}
By Fubini's Theorem,
\begin{eqnarray*}
\E e^{iXY}
  & = & \E \varphi_X(Y) = \int \varphi_X(y) \Q(dy) \\
  & = & \E \varphi_Y(X) = \int \varphi_Y(x) \P(dx)
\end{eqnarray*}
\end{proof}

We can see how Parseval's identity (\ref{eqn:parseval}) might be useful:
on the left-hand side we have the characteristic function $\varphi_X$,
and on the right-hand side we have its probability measure $\P$.
The two sides are linked by another random variable $Y$.
We need to be able to work with both 
its distribution and characteristic function.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Characteristic function of $N(0, 1)$}
% keywords: characteristic function, normal distribution, complex analysis, differential equations, Central Limit Theorem
% end

As it turns out, we will use the normal distribution as that random variable $Y$.
In this section,
we will derive the characteristic function of the standard normal.

Suppose that $X \sim N(0, 1)$.
The key identity:
\begin{equation}
\label{eqn:norm-charfn}
\varphi_X(t) = \E e^{itX} = \int_{-\infty}^\infty \frac{1}{\sqrt{2\pi}} e^{-x^2/2} e^{itx} dx = e^{-t^2/2}
\end{equation}

Here are four ways to prove this identity
(Section 2.3 of Durrett gives two more):

\begin{enumerate}
\item
By real analysis,
for real $\theta$, we can check that
\begin{equation}
\int_{-\infty}^\infty \frac{1}{\sqrt{2\pi}} e^{-x^2/2} e^{\theta x} dx
= e^{\theta^2/2}
\end{equation}

After confirming that the identity holds for $\theta \in \R$,
we can extend it for $\theta \in \C$
by standard complex analysis techniques
such as analytic continuation.
equation~\ref{eqn:norm-charfn} follows by
substituting $\theta = it$.

\item
Use complex line integration.

\item
Use differential equations:
Check that both sides of equation~\ref{eqn:norm-charfn}
satisfy the same differential equation when we differentiate with respect to $t$.
They agree at $t=0$, so they must agree at all $t$.

\item Use the Central Limit Theorem.
This is kind of backwards,
since we usually use characteristic functions to prove the Central Limit Theorem.
But in a previous lecture,
we proved the Central Limit Theorem via Lindeberg's Theorem,
so we're not running in circles.

Take $X_1, X_2, \dots$ i.i.d.~which take on values
of $-1$ and $+1$ with $1/2$ probability each.
Let $S_n = X_1 + \cdots + X_n$.

By the Central Limit Theorem,
$S_n/\sqrt{n} \dcv N(0, 1)$.
Since $x \to e^{itx}$ is a bounded continuous function,
$\E e^{it S_n/\sqrt{n}} \to \E e^{itZ}$,
where $Z \sim N(0, 1)$.
We can compute the left hand side explicitly:
\begin{eqnarray}
\label{norm-charfn-clt}
\E e^{it S_n/\sqrt{n}}
  & = & \left( \E e^{it X_1/\sqrt{n}} \right)^n \quad\text{[because $X_i$'s are i.i.d.]} \\
  & = & \left( \frac{1}{2} e^{it/\sqrt{n}} + \frac{1}{2} e^{-it/\sqrt{n}} \right)^n \\
  & = & \left( \cos (t/\sqrt{n}) \right)^n \\
  & = & \left( 1 - \frac{t^2}{2n} + O \left( \frac{t^4}{n^2} \right) \right)^n \quad\text{[as $n \to \infty$]} \\
  & \to & e^{-t^2/2} \quad\text{[Fact: $(1+x_n/n)^n \to e^x$ if $x_n \to x$]}
\end{eqnarray}
where in the fourth line we used the fact that as $n \to \infty$,
we can approximate $\cos(t/\sqrt{n})$
by its Taylor expansion around $t/\sqrt{n}=0$.

\end{enumerate}

By simple properties of characteristic functions:
\begin{equation}
\varphi_{\mu + \sigma Z}(t) = e^{it\mu} e^{-\sigma^2 t^2/2},
\end{equation}
where $Z \sim N(0, 1)$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Derivation of the inversion formula}
% keywords: Parseval's identity, Fubini's Theorem, convolution, continuous function, weak convergence, almost sure convergence
% end

Now, we are ready to apply Parseval's identity (\ref{eqn:parseval}) with
$Y$ distributed as $\Q = N(0, \sigma^2)$:
\begin{equation}
\label{eqn:parseval-apply}
\int \varphi_X(t) \frac{1}{\sqrt{2\pi\sigma^2}} e^{-t^2/2\sigma^2} dt
=
\int e^{-\sigma^2 x^2/2} \P(dx).
\end{equation}

The left-hand side looks fine,
but we would like to decipher the right-hand side in terms of $\P$.
Let us try to interpret the right-hand side probabilistically.

The following observation can help us:
If $X$ and $Y$ are independent random variables
with X having distribution $\P$ and Y having distribution given by 
density $f_Y$, 
then $X + Y$ has a density described by the following convolution formula
(by Fubini's Theorem):
\begin{equation}
\label{eqn:sum-density}
f_{X+Y}(z) = \int_{-\infty}^\infty f_Y(z-x) \P(dx),
\end{equation}

We want to match the right-hand side of equation~\ref{eqn:parseval-apply}
with equation~\ref{eqn:sum-density}.
Let $Y = Z/\sigma$ be distributed as $N(0, 1/\sigma^2)$ with density
\begin{equation}
\label{eqn:density-y}
f_Y(y) = f_{Z/\sigma}(y) = \frac{\sigma}{\sqrt{2\pi}} e^{-y^2\sigma^2/2}
\end{equation}

By noting that $f_{Z/\sigma}(x) = f_{Z/\sigma}(-x)$,
morph equation~\ref{eqn:sum-density} into:
\begin{equation}
\label{eqn:density-xz0}
f_{X+Z/\sigma}(0) = \int_{-\infty}^\infty f_{Z/\sigma}(x) \P(dx).
\end{equation}

We can now interpret the right-hand side of equation~\ref{eqn:parseval-apply}
as the density of $X+Z/\sigma$ evaluated at 0
times the constant $\frac{\sqrt{2\pi}}{\sigma}$.

Combining equations~\ref{eqn:density-y},
\ref{eqn:parseval-apply}, and \ref{eqn:density-xz0}:
\begin{equation}
\int \varphi_X(t) \frac{1}{\sqrt{2\pi\sigma^2}} e^{-t^2/2\sigma^2} dt
= \frac{\sqrt{2\pi}}{\sigma} \int f_{Z/\sigma}(x) \P(dx)
= \frac{\sqrt{2\pi}}{\sigma} f_{X + Z/\sigma}(0)
\end{equation}

After rearranging:
\begin{equation}
f_{X + Z/\sigma}(0) = \frac{1}{2\pi} \int \varphi_X(t) e^{-t^2/2\sigma^2} dt
\end{equation}

(Note that our calculations above worked
due to two favorable properties of the normal distribution:
first, its characteristic function is proportional to a probability density;
and second, its density is symmetric around 0.

As a matter of aesthetics,
let us give $\sigma$ its usual interpretation as the standard deviation
by replacing $\sigma$ with $1/\sigma$:
\begin{equation}
f_{X + \sigma Z}(0) = \frac{1}{2\pi} \int \varphi_X(t) e^{-t^2 \sigma^2/2} dt
\end{equation}

To get the value of $f_{X + \sigma Z}(x)$ at other values of $x$ other than 0,
note that $f_{X + \sigma Z}(x) = f_{X-x+\sigma Z}(0)$,
which gives us our final formula for the density of $X + \sigma Z$:
\begin{equation}
f_{X + \sigma Z}(x) = \frac{1}{2\pi} \int \varphi_X(t) e^{-itx} e^{-t^2 \sigma^2/2} dt
\end{equation}

In general, the distribution of $X$ could be rough and nasty and not have a density.
Nonetheless, $X + \sigma Z$ does have a density for all $\sigma > 0$,
which is made explicit in terms of $\varphi_X(t)$.
We're almost there!

Now let's try to pinpoint the distribution of $X$ itself.
As $\sigma \to 0$, $X + \sigma Z \ascv X$,
so therefore also in distribution.
Thus for all bounded continuous functions $g$,
\begin{equation}
\label{eqn:gx-limit}
\E g(X) = \lim_{\sigma \to 0} \E g(X + \sigma Z)
= \lim_{\sigma \to 0} \int g(x) f_{X + \sigma Z}(x) dx
\end{equation}

Therefore, $\varphi_X(t)$ determines the distribution of $X$
as the weak limit of the distributions of $X + \sigma Z$ as $\sigma \to 0$.

We can do more when $\varphi_X(t)$ is absolutely integrable,
that is, if $\int |\varphi_X(t)| dt < \infty$.
This condition basically imposes some smoothness constraints on
the distribution of $X$.
By the Dominated Convergence Theorem,
we can swap the limit and the expectation in equation~\ref{eqn:gx-limit}:
\begin{equation}
\E g(X)
= \lim_{\sigma \to 0} \int g(x) f_{X + \sigma Z}(x) dx
= \int g(x) f_{X}(x) dx,
\end{equation}
where $X$ now has an explicit continuous density:
\begin{equation}
f_X(x) = \frac{1}{2\pi} \int \varphi_X(t) e^{-itx} dt
\end{equation}

Durrett gives a general inversion formula
regardless of whether $X$ has a density or not.
If $a < b$, then
\begin{equation}
\label{eqn:inversion-durrett}
\lim_{T \to \infty} \frac{1}{2\pi} \int_{-T}^T
\frac{e^{-ita} - e^{-itb}}{it} \varphi_X(t) dt
= \P(a, b) + \frac{1}{2} \P(\{a, b\})
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Examples}
% keywords: exponential distribution, Cauchy distribution, Laplace transform
% end

\begin{example}
\upshape Let $X \sim \text{exp}(\lambda)$
(an exponential distribution with parameter $\lambda$)
with density $f(x) = \lambda e^{-\lambda x}$.
We can easily compute its Laplace transform:
%
\[\E[e^{-\theta X}]=\int_{0}^{\infty}e^{-\theta x}
f(x)dx=\frac{\lambda}{\lambda + \theta} \]
%
Using some facts of complex analysis we
can evaluate the characteristic function by putting $\theta = -it$.
%
\[\E[e^{itX}]=\frac{\lambda}{\lambda -it} \]
%

Note that if the characteristic function of a random variable $X$ is
real then $X\stackrel{\scriptscriptstyle d}{=}-X$ and also
$\varphi_{-X}(t)=\E[e^{-itX}]=\varphi_{X}(t)$. So:
%
\[X\stackrel{\scriptscriptstyle d}{=}-X \Leftrightarrow
\varphi_X(-t)=\varphi_{-X}(t)  \Leftrightarrow
\varphi_X(t)=\overline{\varphi_{X}(t)} \] 
%
Now look at the characteristic
function of $X-\hat{X}$,
where $\hat X\sim \exp(\lambda)$ is
independent of $X$.
%
\[\E [e^{it(X-\hat X)}]=\frac{\lambda}{\lambda -it}\frac{\lambda}{\lambda
+it}=\frac{\lambda^2}{\lambda^2 +t^2}\]
%
Take $\lambda=1$.
It's easy to find out the density of $X-\hat{X}$.
%
\[f_{X-\hat{X}}(x)=\frac{1}{2}e^{-|x|}, \qquad x \in \R \]
%
So we have learned a classic integral
%
\[\int_{-\infty}^{\infty}e^{itx}\underbrace{\frac{1}{2}e^{-|x|}}_{\text{density}}dx=\underbrace{\frac{1}{1+t^2}}_{\text{Characteristic
    function at t}} \]
%

Also, since $\varphi(t)=\int_{-\infty}^{\infty}\frac{1}{1+t^2}dt < \infty$,
the inversion formula gives us:
%
\begin{equation}\label{eq:chachyinv}
\frac{1}{2\pi}\int_{-\infty}^{\infty}e^{-itx}\frac{1}{1+t^2}dt=\frac{1}{2}
e^{-|x|}
\end{equation}
%
Take $x=0$.  We learn another classic integral
\[\int_{-\infty}^{\infty}\frac{1}{1+t^2}dt=\pi\]
%

We continue our discussion in the next example.
\end{example}

\begin{example}[Cauchy processes]
\upshape Let $Y_1$ be a random variable with the Cauchy distribution.
Then the probability measure of $Y_1$ is given by
\[
P(Y_1 \in dx) = \frac{dx}{\pi (1+x^2)}.
\]
Notice that $E|Y_1| = \infty$ and the Cauchy distribution has a
heavy tail compared to other distributions. Using the inversion
formula \eqref{eq:chachyinv}, the characteristic function of $Y_1$
is computed as
\[
\varphi(t) = \E(e^{i t Y_1}) = e^{-|t|}.
\]
Now let $Y_1, \ldots, Y_n$ be i.i.d.
with the Cauchy distribution and $S_n = Y_1 + \cdots + Y_n$.
Then the characteristic function of $S_n$ is
\begin{eqnarray*}
\E \Big( e^{i t S_n/n} \Big) & = & \prod_{i=1}^n \E \Big( e^{i t Y_i/n} \Big) \\
& = & \prod_{i=1}^n e^{-|\frac{t}{n}|}\\
& = & e^{-|t|}.
\end{eqnarray*}
Hence $S_n/n \deq Y_i \deq Y_1$.

This is the amazing property of the Cauchy distribution:
the average of $n$ independent Cauchy variables
has the same invariant Cauchy distribution.
There is no law of large numbers effect here!
The reason is that since $E|Y_i| = \infty$,
the law of large numbers does not apply.
\end{example}

\bibliographystyle{plain}
\bibliography{../books}

\end{document}